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The paper aims to investigate the achievements of the theoretical and practical basis of environmental 
policy analysis in order to study their works and point out the future possible research direction. It sorts 
out researches about environmental efficiency assessment and reviews the works about the theory and 
application of efficiency analysis around the world. It is suggested that environmental efficiency 
evaluation theory under small samples and DEA method with undesirable outputs will further extend 
the research on environmental efficiency evaluation. Also, this review confirms that more studies in 
methods and their applications in this area are in urgent need. 
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1. Introduction 


Environmental issues have become one of the most important 
problems related with social and economic sustainable develop- 
ment. Evaluating environmental efficiency in different regions 
and sectors has strong practical implications. This work not only 
benefits for people understand the difference among their envir- 
onmental performances, but also provides an objective reference 
point for improving environmental performances. However, the 
current evaluation methods for environmental efficiency are 
mostly based on the determination of the inputs and outputs 
evaluation index. Some evaluation methods (such as the data 
envelopment analysis, Stochastic Frontier Analysis) are proposed 
based on the macro data or micro data. Finally, the environmental 
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efficiency is measured. Because environmental efficiency evalua- 
tions are very complex, the negligence of application scenarios 
and invariably using some specific traditional evaluation methods 
may lead to the following disadvantages in practice: (1) failure to 
take into account environmental efficiency for small samples 
results in imprecise evaluation; (2) the selection of the variables 
has a great deal of subjectivity; and (3) some curing and non- 
dynamic evaluation methods can not deal with the change of 
socio-economic environment. Solving these problems is impor- 
tant for the development of environmental efficiency evaluation 
methods. 

Now, measuring environment efficiency has become an essen- 
tial direction in research. So far, scholars have proposed several 
quantitative models to solve the complex environmental pro- 
blems [1]. More and more people have generally recognized the 
importance of environmental efficiency evaluation because it can 
provide designers and public policy makers with quantitative 
information for performance evaluation, policy analysis and public 
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communication. All of these benefits will make the decision of 
environmental policy-making more scientific, empirical and sys- 
tematic than before. 

So far, there have been many quantitative analysis techniques 
in environmental efficiency evaluation, among which the produc- 
tion efficiency analysis has drawn widespread interests in recent 
years. Modern production efficiency analysis began to be applied 
in the study of environmental problems in 1980s, and with 
more concerns on the environmental problem. The efficiency 
evaluation method is widely used in environmental evaluation 
systems [2]. Data envelopment analysis (DEA) is an effective non- 
parametric method for evaluating the relative effectiveness of the 
decision making units (DMUs) [3,4]. The exact functional relation- 
ship (refer to function formula) between inputs and outputs need 
not be considered in DEA technology. DEA does not need decision 
maker to provide the information on weights. The weights can be 
gained through a programming, that is, no pre-estimated para- 
meters are needed. Therefore, the weights can avoid being 
subjectivity. But the traditional DEA efficiency model only con- 
siders the desirable outputs while neglecting the undesirable 
outputs in the actual production process. In fact, the undesirable 
output is usually produced companied with the desirable outputs. 
For example, in a thermal power plant, it is inevitable that 
undesirable outputs will be generated in the power generation 
process, such as the emissions of carbon dioxide. The output 
maximization assumption of the traditional DEA efficiency model 
can not be applied for this scenario. Therefore, how to consider 
the undesirable outputs in the traditional DEA efficiency model 
becomes a topic with great theoretical significance and applica- 
tion value. To deal with this issue, many scholars have worked on 
the environmental efficiency and gained a lot of achievements 
[5-11]. Concluding the current works and pointing out some 
interesting and valuable directions for measuring the environ- 
mental efficiency are urged, so this review is believed to be 
necessary and timely. 

The rest of this paper is structured as follows. Section 2 
reviews major related researches. Section 3 introduces the DEA 
models with undesirable output and classifies them into several 
parts. Finally, concluding remarks are given in Section 4. 


2. Origin of environmental efficiency evaluation 


Comprehensive evaluation on the environmental efficiency 
traces to the time when researchers focused on the limited energy 
and the carbon dioxide emissions generated in the production. 
By simulating the carbon dioxide emissions scenario, early 
researchers provided some advice for energy policy in the future. 
Edmonds and Reilly thought that the establishment of a global 
energy environment evaluation model is very important for 
energy analysis and environmental decision-making, and they 
proposed the global energy and economic development model 
which can predict changes for the next one hundred years [12]. 
Additionally, they gave the specific structure of the model and 
explained the results of this model. By analyzing the quantitative 
relationship between the global natural gas distribution and 
carbon dioxide, Reister proposed that the main determinant of 
carbon dioxide emissions was the energy supplier, so the key to 
controlling carbon dioxide emissions is to control its source [13]. 
Harvey believed that the increase of carbon dioxide concentration 
would promote the effects of photosynthesis. Based on this, he 
developed a formula to analyze the impact that the carbon 
dioxide volume fraction on the carbon preservation parameters 
of photosynthesis, and further studied the global carbon cycle 
model through simulating carbon dioxide fertilization effect [14]. 
Gustavsson et al. indicated the government can establish efficient 


energy end-use technologies, cogeneration and a recycling econ- 
omy through the implementation of policies. The energy system 
can still reduce carbon dioxide emissions by 75% through end-use 
technologies, cogeneration and circular economy without increas- 
ing water and power consumption under sustaining economic 
growth [15]. Kamiuto built a simple model including the air, the 
biosphere and the hydrosphere to describe the global carbon 
cycle, and he found that since the chaos deforestation and the 
changes in land use around 1875, the original “big tank” which 
absorbed carbon dioxide has reduced or disappeared, thus chan- 
ging the carbon dioxide emission rate [16]. Svendsen took Den- 
mark as an example and proposed that carbon trading should be 
introduced into plants in the private sector, and management 
department of the public power sector should be built, in order to 
reduce carbon dioxide emissions by 25% by 2005; Meanwhile, the 
carbon tax would reach US$50 per family, the transportation 
sector and private plants were not included at that time [17]. 
Yang and Zhang introduced five approaches of calculation meth- 
ods of carbon dioxide emission from bio-energy utilization, which 
were summarized from the perspective of resources and utiliza- 
tion [18]. However, most of the above works are qualitative. The 
quantitative evaluation for environment system with multiple 
inputs and outputs is badly needed as the impact factors on 
carbon dioxide emissions compose a complex system. 

Soon after, some researchers have begun to pay attention to 
reducing greenhouse gas emissions, and have presented some 
multi-objective decision analysis models and Integrated Assess- 
ment of Climate Protection Strategies (ICLIPS) evaluation meth- 
ods. Fare et al. introduced environmental efficiency variables 
based on the decomposition of total factor productivity in the 
pollution variables and input-output efficiency variables, and 
formed the environmental efficiency variables, giving the calcula- 
tion methods as well [7]. Diesendorf built models considering 
ethical principles, target groups, evaluation and implementation 
strategy, in which all levels of government, business and com- 
munity organizations in collaborative efforts are involved. He 
indicated that the most important thing is to speed up commu- 
nity participation and empowerment in order to form a better 
atmosphere to promote the implementation of sustainable devel- 
opment policies and coordination of environmental protection 
and socio-economic development [19]. Zaim and Taskin estab- 
lished the environmental benefits variables using a non-para- 
metric method for each OECD member country, and measured the 
output they sacrificed to achieve better environmental benefits 
[20]. Voorspools et al. investigated several types of possible 
indirect sources of greenhouse gases, and used life cycle assess- 
ment to measure two different types [21]. Their empirical results 
suggested that nuclear power produces minimal greenhouse gas 
emissions, followed by wind power in coastal areas, and the worst 
among the three was photovoltaic power. Matthews designed an 
evaluation method for bio-fuel production systems of energy and 
carbon budget [22]. He thought that further research should 
validate the input assumptions and evaluate the integrity and 
utility of the budget, as well as the energy generation systems. 
Sands and Leimbach also proposed core methods for Integrated 
Assessment of Climate Protection Strategies (ICLIPS), in which 
they use agriculture and land-use models, using the carbon 
emissions evaluation program where the land use changes were 
the main cause for the overall greenhouse gas emissions [23]. In 
their simulation, land-use changes significantly affected carbon 
emissions. Pasurka calculated the changes in nitrogen oxides and 
sulfur dioxide emissions with the inputs changes of technology, 
fuel and non-fuel by using the new decomposition model and the 
distance function. Besides, he suggested that the major emissions 
of sulfur dioxide decreased with the changes of output groups, 
while nitrogen oxides decreased along with the fuel consumption 
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and the changes of output groups [24]. Manan et al. introduced a 
framework for the implementation of a Malaysian energy effi- 
ciency award, which is proposed based on the survey conducted 
in their research, and on various models of energy award 
implementation around the world [25]. Although these studies 
have promoted the optimization for environmental policies in a 
certain degree in a number of countries and regions, researchers 
have also come to realize that the sustainability of energy 
conservation and environmental protection is largely interactive 
with the efficiency increase in the production process, which 
means that the economic efficiency with environmental consid- 
erations need to be studied [1]. 

The impact factors of environmental efficiency include the 
total profit of resources consumption, its total investment, profits, 
the amount of resources, the ratio of GDP to energy consumption, 
environmental performance and resource production. Apart from 
that, technology, utility, supply, demand, and staff and resource 
availability are also important factors in the constraints of 
environmental efficiency. Meanwhile, as for the objective evalua- 
tion of environmental efficiency, we must also consider the most 
important factor—the availability of sample data, and avoid the 
query of the reliability of the quantitative evaluation for environ- 
mental efficiency. Because there are few studies on environmen- 
tal efficiency analysis and DEA is a very powerful tool to measure 
the efficiency, this paper will focus on the DEA efficiency models 
considering undesirable outputs and the analysis on these envir- 
onmental models. In the following sections, we will review the 
efficiency analysis theory and its applications, and explore the 
achievements of these results to provide theoretical foundation 
and objective basis for environmental policy analysis and design 
as well as its possible deficiencies. 


3. DEA model considering undesirable outputs 


As one of the main tools of the production process efficiency 
analysis, the DEA technique has drawn more and more research- 
ers’ attention since it was proposed by Charnes et al. [26]. Now it 
has received great developments over the past thirty years 
[26,27]. Many researches focus on its applications or its theory 
developments, especially theoretical aspect. Many DEA methods 
are proposed, such as the models of efficiency measure, imposi- 
tion of restrictions on the weight method, investigation of vari- 
able characteristics and modeling aspects of data transformation, 
etc. Environmental efficiency as an interesting topic has both 
theoretical value and practical meaning. So far the researchers 
have achieved a great deal of achievements in both aspects of 
environmental efficiency evaluation. 

During the production process under normal circumstances, 
undesirable outputs will be inevitably produced, such as a variety 
of environmental pollutants. This situation can not meet the 
“maximum outputs” hypothesis of traditional DEA efficiency 
model, so undesirable outputs must be specially treated to 
expand traditional DEA efficiency model [28]. Taking undesirable 
outputs into the technical efficiency evaluation framework, we 
must minimize resource inputs while reducing emissions of 
undesirable outputs under the circumstance of keeping the 
product output unchanged, or maintain the same input target 
while increasing the desirable outputs and reducing undesirable 
outputs as much as possible. 

DEA efficiency models considering undesirable outputs can be 
divided into three categories. The first category is taking the 
undesirable outputs as inputs for processing [29,30], but it can 
not reflect the real production process in some degree [11]. The 
second category is conducting data transformation to undesirable 
outputs first, and then evaluating the environmental efficiency by 


using the traditional efficiency model based on the transformed 
data [11,31]. Due to strong convexity constraints, it can only be 
solved under variable returns to scale. The third category is 
considering the disposability of production technology into non- 
parametric DEA model, which is produced by Fare et al. [5,6,9,32], 
and then further extended by more studies [33-36]. Meanwhile, 
Rovere et al. suggested that a new approach is urgently needed to 
takes into account the technical, socioeconomic, environmental 
and technological factors of the various alternatives for sector 
expansion at the same time. They also indicated that multi- 
criteria analysis may be a good evaluation tool [37]. Although 
practical value of this approach is verified, this approach only 
drew a few scholars’ attention [38]. 

Among many measures methods, the most common one for 
evaluating the efficiency is the radial efficiency measure, which is 
reciprocal to the Shephard’s distance function. More generalized 
distance function also is formed in DEA models based on the 
traditional Shephard’s distance function [39]. Besides the above 
two, the hyperbolic efficiency measure can increase the desirable 
outputs and reduce undesirable outputs [5]. The non-radial and 
slack-variables-based efficiency measure considering undesirable 
outputs in the non-parametric DEA efficiency model framework 
bring a new flavor to this problem [40,41]. 

With the international community’s focus on the environment 
and sustainable development issues, it is very necessary to integrate 
the environmental factors into the study of production efficiency. 
The non-parameter DEA model has been widely applied in environ- 
mental efficiency assessment research because DEA does not need a 
functional form of the production frontier before evaluation and has 
great flexibility. Moreover, this method provides the measure for 
improving the performance of decision making units [2]. Fare et al. 
first deals with pollution variables by using weak disposability of the 
inputs and outputs. Weak disposability assumes the change of 
undesirable outputs definitely affects the other outputs, that is, if 
you want to reduce pollution and other undesirable outputs, some 
desirable outputs must be given up as a result [5]. 

Since Färe et al. proposed the first DEA model to deal with 
undesirable outputs with the weak disposability, the environ- 
mental efficiency evaluation considering the undesirable outputs 
were largely extended [5]. Allen, Sarki and Talluri had summar- 
ized the applications of the DEA model in environmental effi- 
ciency research [42,43]. Lansink and Reinhard took pollutants as 
input variables and explored the technical efficiency and potential 
technology growth of Dutch pig farms based on weak-treatment 
DEA model [44]. Li and Cheng suggested dealing with desirable 
and undesirable outputs simultaneously in the DEA model, and 
built a new DEA model for analyzing forty-six city industrial 
sectors productivity of China [45]. Kordrostami and Amirteimoori 
proposed the environmental efficiency evaluation of decision- 
making units with chain internal structure, and provided a 
corresponding efficiency evaluation index based on the multi- 
component efficiency evaluation method [46]. Murty et al. esti- 
mated environmental efficiency; Malmquist produced an index 
and a shadow price of pollutants of the Indian sugar industry, 
based on Weak and Strong Disposability of Bad Outputs [47]. 
Brannlund et al. and Gomes and Lins also developed a carbon 
emissions quota model based on the non-expected output and 
non-parametric DEA efficiency model [48,49]; Zhou et al. pro- 
posed carbon emissions decomposition analysis method based on 
the environmental DEA technique and Shephard’s distance func- 
tion [34]. Researchers have developed a variety of DEA efficiency 
models considering undesirable outputs, including environmental 
performance evaluation, environmental regulation impact assess- 
ment, pollutant emission quota, shadow prices of pollutants 
estimation and other environmental systems assessment 
problems. Some results are also related to the following aspects: 
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The first method is taking undesirable outputs as investments 
[30,50-52]. Environmental pollutants and other undesirable out- 
puts are often associated with resources invested in the produc- 
tion process, increased and reduced with desirable outputs, which 
has similar relationship with the traditional production function 
between inputs and outputs. In the efficiency evaluation process, 
deal with undesirable outputs as an input indicator added to the 
DEA model, and take this model to analyze the environmental 
efficiency of decision making units. Haynes et al. proposed an 
environmental protection production frontier to research the 
environmental efficiency based on data envelopment [52]. Rein- 
hard et al. analyzed the environmental efficiency of Dutch dairy 
farms with DEA, and compare with the random production 
frontier evaluation methods [53]. Hailu and Veeman analyzed 
the environmental efficiency of the Canadian paper industry using 
DEA methods [30]. 

Second one is data transfer function method [11,28,54]. The 
data transfer function method considers the pollutants as outputs, 
and at the same time, transfers the-smaller-the-better undesir- 
able outputs into the-bigger-the-better desirable output, and then 
takes the transformed output as an ordinary output, analyzing 
environmental efficiency of decision making units with the 
traditional DEA model. There are three data transfer function 
forms: the negative output method, the linear data transfer 
method [11] and the non-linear data transfer method [28,54]. 

Third one is distance function method. Chung took pollutants 
into the technical efficiency analysis framework and proposed a 
DEA Environmental Efficiency model based on distance function 
and weak handling of pollutants by which a especially direction is 
set [39]; when the efficiency of the decision making unit is 
improved along the direction, one can increase desirable outputs 
while reducing undesirable outputs properly. By this method, the 
defect of the above two methods, i.e., a one-dimensional change, 
can be made up, such as not reflecting the production process, 
undermining the convex of production possibility set, etc. This 
method breaks through the traditional method of radial measure 
of efficiency improvement and has certain superiority. 


4. Conclusions 


In this paper, we review the current studies on the environ- 
mental efficiency evaluation. The background why this problem 
draws more and more researchers’ attention is introduced in 
details. More importantly, the route of this direction is firstly 
illustrated. Then more works are focused on the non-parametric 
DEA technique with undesirable output because this approach 
has a lot of advantages on evaluating the environmental effi- 
ciency. Now, many achievements have done on this area. 

However, the non-parametric DEA efficiency model is essen- 
tially a non-random technique. Traditional DEA has some defi- 
ciencies. For example, it does not consider the uncertainty of the 
data, it can not make statistical inferences for the efficiency 
analysis results, and it does not give an obvious statistical 
significance of the evaluation value. 

Studies that consider statistical properties of undesirable out- 
puts of the non-parametric DEA efficiency model are rare, and 
achievement of environmental efficiency evaluation based on small 
samples is even more rare [55]. All these factors mean that there 
are many Statistical defects both in considering undesirable out- 
puts DEA method itself and improving analysis of the environ- 
mental efficiency quantitative evaluation under small samples 
conditions [56,57]. Therefore, the following aspects of conducting 
research will further promote the depth and breadth of considering 
non-expected output environmental efficiency evaluation method: 


Research on evaluation theory and method based on small 
samples. How to improve the evaluation accuracy of environ- 
mental efficiency under small sample condition is an issue to be 
addressed. Under the circumstances of small overall sample size 
or unknown distribution error term of traditional modeling 
methods, the statistical inference only based on original sample 
data is not credible; when using the evaluation system model 
based on these, the evaluation precision is not satisfying. So we 
must take full account of defects from the perspective of informa- 
tion disclosure on environmental protection, analyze the char- 
acteristics of environmental efficiency evaluation under small 
samples circumstances, and establish environmental efficiency 
evaluation models by using a new method, namely applying 
V-fold and Bootstrap. It is also needed to determine the optimal 
value of V, and to test the effectiveness of parameter t value 
hypothesis of small samples based on the Bootstrap method. 

Research on environmental efficiency evaluation theory and 
method based on undesirable outputs DEA. In order to achieve a 
more accurate quantitative evaluation of environmental effi- 
ciency, we need to establish a series of new DEA models based 
on studying specific characters of environmental efficiency eva- 
luation, including: the efficiency measure type aspect, using a 
series of non-radial DEA models to study undesirable outputs of 
environmental efficiency evaluation different from the traditional 
approach; building two-goal DEA model of environmental effi- 
ciency, taking into account the desirable outputs increasing and 
undesirable outputs reducing in the model, and solving the 
model. On the basis of these studies, non-radial-two-goal DEA 
environmental efficiency model will be presented considering 
undesirable outputs, and these models will be solved by appro- 
priate methods and techniques, so we can get closer to the actual 
evaluation results of environmental efficiency. Then, by putting 
random factors into the common non-parametric-efficiency DEA 
model considering undesirable outputs, analyzing the influence of 
environmental efficiency evaluation results by these random 
factors and statistical noise is then possible. 

Research on environmental efficiency evaluation method and 
its applications on the condition of small samples. When current 
environment-related data is of insufficient disclosure, the char- 
acters of environmental efficiency evaluation system is similar to 
the characters of small sample modeling method combined with 
the DEA model considered undesirable outputs. This requires a 
proper combination of small sample optimization and DEA model 
considering undesirable output. The evaluation model and eva- 
luation system should be established and then corresponding 
application should be carried out in order to guide the future 
environmental efficiency evaluation. 
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